Break, Learn, Refine – The Art of Hypothesis-Driven Development of ML-Powered Search

Uzum Market is a rapidly growing e-com in Uzbekistan with more than 500k items available. It’s impossible to navigate such a multi-categorical catalog without a large-scale search system, thus making it a vital technology for the marketplace.

It includes many parts: sparse retrieval, spelling correction, typing suggestions, linear and gradient boosting ML models for ranking. Neural retrieval is about to join the gang. We started with a much simpler pipeline and we’ve been improving the search continuously ever since. We’d like to share with the community mistakes we’ve made along the way, principles we deduced from them, and what kind of rails were built to streamline hypothesis testing.

We’ll dive into:

• Metrics and evaluation. Why our initially chosen offline metrics for search relevance turned out to be completely wrong and how we determined the right ones
• Targets for LTR models. What kind of clickstream data we are using and why the attribution modeling is crucial
• AB testing

Andrey Kulagin

Uzum Market

Andrey is the Head of Machine Learning at Uzum Market. He is an expert in e-commerce product search, with a track record of establishing ML-driven search systems in 2 online marketplaces. Manages ML teams specializing in search optimization, recsys, demand forecasting, and uplift modeling.